Papers with automated scoring systems
Atypical Inputs in Educational Applications (N18-3)
Copied to clipboard
| Challenge: | atypical characteristics of some responses make it difficult for an automated scoring system to assign a valid score . a typical spoken response with a lot of background noise may suffer from frequent errors in automated speech recognition . |
| Approach: | They propose a pipeline that detects and processes non-scorable responses at run-time . they also propose linguistic filtering models for spoken responses in language tests . |
| Outcome: | The proposed pipeline detects and processes non-scorable responses at run-time and evaluates them for spoken responses in language proficiency assessment. |
Automated Scoring: Beyond Natural Language Processing (C18-1)
Copied to clipboard
| Challenge: | In this paper, we argue that building operational automated scoring systems is a task that has disciplinary complexity above and beyond competitive shared tasks. |
| Approach: | They argue that building operational automated scoring systems is a task that has disciplinary complexity above and beyond standard competitive shared tasks . they argue that it is essential for us as NLP researchers to understand and incorporate these perspectives in our research and work towards a mutually satisfactory solution . |
| Outcome: | The proposed approach is based on the findings of a recent conference on automated scoring. |
Calibrating LLMs with Preference Optimization on Thought Trees for Generating Rationale in Science Question Scoring (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for generating rationales that justify scoring decisions are not accurate and often contain hallucinated information. |
| Approach: | They propose a framework capable of generating more faithful rationales and matching performance with classifier-based scoring systems. |
| Outcome: | The proposed framework achieves 38% improvement in QWK score compared to prior work . it can be used to match performance with classifier-based scoring systems . |